{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "punL79CN7Ox6"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "_ckMIh7O7s6D"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CH5gnvxl-N3U"
      },
      "source": [
        "# What's in a (sub)word?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S5Uhzt6vVIB2"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_for_deep_learning/l09c06_nlp_subwords.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_for_deep_learning/l09c06_nlp_subwords.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ykxAKKa1Dl0s"
      },
      "source": [
        "In this colab, we'll work with subwords, or words made up of the pieces of larger words, and see how that impacts our network and related embeddings."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QQCr_NAT-g5w"
      },
      "source": [
        "## Import TensorFlow and related functions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Q8Wa_ZlX-mPH"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MRHk-4Te-yLJ"
      },
      "source": [
        "## Get the original dataset\n",
        "\n",
        "We'll once again use the dataset containing Amazon and Yelp reviews. This dataset was originally extracted from [here](https://www.kaggle.com/marklvl/sentiment-labelled-sentences-data-set)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XJAxrOLi-02C"
      },
      "outputs": [],
      "source": [
        "!wget --no-check-certificate \\\n",
        "    https://drive.google.com/uc?id=13ySLC_ue6Umt9RJYSeM2t-V0kCv-4C-P \\\n",
        "    -O /tmp/sentiment.csv"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Dr-EDUKP_HBl"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "\n",
        "dataset = pd.read_csv('/tmp/sentiment.csv')\n",
        "\n",
        "# Just extract out sentences and labels first - we will create subwords here\n",
        "sentences = dataset['text'].tolist()\n",
        "labels = dataset['sentiment'].tolist()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8zut9Wng_R3B"
      },
      "source": [
        "## Create a subwords dataset\n",
        "\n",
        "We can use the existing Amazon and Yelp reviews dataset with `tensorflow_datasets`'s `SubwordTextEncoder` functionality. `SubwordTextEncoder.build_from_corpus()` will create a tokenizer for us. You could also use this functionality to get subwords from a much larger corpus of text as well, but we'll just use our existing dataset here.\n",
        "\n",
        "The Amazon and Yelp dataset we are using isn't super large, so we'll create a subword `vocab_size` of only the 1,000 most common words, as well as cutting off each subword to be at most 5 characters.\n",
        "\n",
        "Check out the related documentation [here](https://www.tensorflow.org/datasets/api_docs/python/tfds/features/text/SubwordTextEncoder#build_from_corpus)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aElsgxia_43g"
      },
      "outputs": [],
      "source": [
        "import tensorflow_datasets as tfds\n",
        "\n",
        "vocab_size = 1000\n",
        "tokenizer = tfds.deprecated.text.SubwordTextEncoder.build_from_corpus(sentences, vocab_size, max_subword_length=5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0XNZWGKqBDc3"
      },
      "outputs": [],
      "source": [
        "# Check that the tokenizer works appropriately\n",
        "num = 5\n",
        "print(sentences[num])\n",
        "encoded = tokenizer.encode(sentences[num])\n",
        "print(encoded)\n",
        "# Separately print out each subword, decoded\n",
        "for i in encoded:\n",
        "  print(tokenizer.decode([i]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gYnbqctXGKcC"
      },
      "source": [
        "## Replace sentence data with encoded subwords\n",
        "\n",
        "Now, we'll re-create the dataset to be used for training by actually encoding each of the individual sentences. This is equivalent to `text_to_sequences` with the `Tokenizer` we used in earlier exercises."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rAmql34aGfeV"
      },
      "outputs": [],
      "source": [
        "for i, sentence in enumerate(sentences):\n",
        "  sentences[i] = tokenizer.encode(sentence)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jNnee_csG5Iz"
      },
      "outputs": [],
      "source": [
        "# Check the sentences are appropriately replaced\n",
        "print(sentences[1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zpIigjecHVkF"
      },
      "source": [
        "## Final pre-processing\n",
        "\n",
        "Before training, we still need to pad the sequences, as well as split into training and test sets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "INIFSAcEHool"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "max_length = 50\n",
        "trunc_type='post'\n",
        "padding_type='post'\n",
        "\n",
        "# Pad all sentences\n",
        "sentences_padded = pad_sequences(sentences, maxlen=max_length, \n",
        "                                 padding=padding_type, truncating=trunc_type)\n",
        "\n",
        "# Separate out the sentences and labels into training and test sets\n",
        "training_size = int(len(sentences) * 0.8)\n",
        "\n",
        "training_sentences = sentences_padded[0:training_size]\n",
        "testing_sentences = sentences_padded[training_size:]\n",
        "training_labels = labels[0:training_size]\n",
        "testing_labels = labels[training_size:]\n",
        "\n",
        "# Make labels into numpy arrays for use with the network later\n",
        "training_labels_final = np.array(training_labels)\n",
        "testing_labels_final = np.array(testing_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QC9A-sTpPPiL"
      },
      "source": [
        "## Train a Sentiment Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eDKcL64IPcfy"
      },
      "outputs": [],
      "source": [
        "embedding_dim = 16\n",
        "model = tf.keras.Sequential([\n",
        "    tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),\n",
        "    tf.keras.layers.GlobalAveragePooling1D(),\n",
        "    tf.keras.layers.Dense(6, activation='relu'),\n",
        "    tf.keras.layers.Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VqkMNtIeP3oz"
      },
      "outputs": [],
      "source": [
        "num_epochs = 30\n",
        "model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])\n",
        "history = model.fit(training_sentences, training_labels_final, epochs=num_epochs, \n",
        "                    validation_data=(testing_sentences, testing_labels_final))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sj18M42kQkCi"
      },
      "source": [
        "## Visualize the Training Graph\n",
        "\n",
        "We can visualize the training graph below again. Does there appear to be a difference in how validation accuracy and loss is trending compared to with full words?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uy8KIMPIQlvH"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "\n",
        "def plot_graphs(history, string):\n",
        "  plt.plot(history.history[string])\n",
        "  plt.plot(history.history['val_'+string])\n",
        "  plt.xlabel(\"Epochs\")\n",
        "  plt.ylabel(string)\n",
        "  plt.legend([string, 'val_'+string])\n",
        "  plt.show()\n",
        "  \n",
        "plot_graphs(history, \"accuracy\")\n",
        "plot_graphs(history, \"loss\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_m7QzouQQ1Rs"
      },
      "source": [
        "## Get files for visualizing the network\n",
        "\n",
        "Once again, you can visualize the sentiment related to all of the subwords using the below code and by heading to http://projector.tensorflow.org/ to upload and view the data.\n",
        "\n",
        "Note that the below code does have a few small changes to handle the different way text is encoded in our dataset compared to before with the built in `Tokenizer`.\n",
        "\n",
        "You may get an error like \"Number of tensors (999) do not match the number of lines in metadata (992).\" As long as you load the vectors first without error and wait a few seconds after this pops up, you will be able to click outside the file load menu and still view the visualization."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dezs4wE5RMQu"
      },
      "outputs": [],
      "source": [
        "# First get the weights of the embedding layer\n",
        "e = model.layers[0]\n",
        "weights = e.get_weights()[0]\n",
        "print(weights.shape) # shape: (vocab_size, embedding_dim)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LXKqy9Z1RSmt"
      },
      "outputs": [],
      "source": [
        "import io\n",
        "\n",
        "# Write out the embedding vectors and metadata\n",
        "out_v = io.open('vecs.tsv', 'w', encoding='utf-8')\n",
        "out_m = io.open('meta.tsv', 'w', encoding='utf-8')\n",
        "for word_num in range(0, vocab_size - 1):\n",
        "  word = tokenizer.decode([word_num])\n",
        "  embeddings = weights[word_num]\n",
        "  out_m.write(word + \"\\n\")\n",
        "  out_v.write('\\t'.join([str(x) for x in embeddings]) + \"\\n\")\n",
        "out_v.close()\n",
        "out_m.close()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "v04wBMybRoGx"
      },
      "outputs": [],
      "source": [
        "# Download the files\n",
        "try:\n",
        "  from google.colab import files\n",
        "except ImportError:\n",
        "  pass\n",
        "else:\n",
        "  files.download('vecs.tsv')\n",
        "  files.download('meta.tsv')"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "l09c06_nlp_subwords.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
